[v7r2] Move to a src/ layout by chrisburr · Pull Request #4858 · DIRACGrid/DIRAC

chrisburr · 2020-12-07T15:17:16Z

The first step in implementing what was discussed at the last BiLD. Obviously this PR touches a lot of files but it should be quite easy to review as there are very few actual changes. It is split into three commits:

79d6af7: This is the big diff which is simply moving the files:

git mv __init__.py AccountingSystem ConfigurationSystem Core DataManagementSystem FrameworkSystem Interfaces MonitoringSystem ProductionSystem RequestManagementSystem Resources ResourceStatusSystem StorageManagementSystem TransformationSystem Workflow WorkloadManagementSystem src/DIRAC/

7fa80a7: Fixes required to get the CI passing again
5fb66e6: Fixes for some pylint errors which show up in the certification test files

First part of #4774

BEGINRELEASENOTES

*Python 3
NEW: Moved to a src/ repository layout

ENDRELEASENOTES

chrisburr · 2020-12-07T16:32:20Z

          #     gracefully, make them non-daemonic and use a suitable
          #     signalling mechanism such as an Event."
          - pytest --no-cov -k 'not test_BaseType_Unicode and not test_nestedStructure and not testLockedClass'
+          # - pytest --no-cov src/DIRAC/Core/Security/test


@chaen In the previous Python 3 merge request I had missed this test. I tried adding it here but it fails in a non obvious way. Would you be able to take a look at some point?

(if you can you should work on top of this branch as I fixed all except one of the failures already)

Ah you mean the Security tests ? Yes, these are nasty, because they unload some modules from memory in order to do a pyGSI/M2Crypto comparison. I think they can be simplified, and then not run anymore as a specific tests. I'll have a look !

So that seems to be an M2Crypto issue. I have opened a bug report. I suspect that the problem lies in the C-API, probably some mess with str/bytes but since I have no knowledge of that, I will wait to see if there is a reaction before embarking in fixing it myself

https://gitlab.com/m2crypto/m2crypto/-/issues/287

fstagni · 2020-12-07T17:44:51Z

I have seen you moved tests/Utilities → src/DIRAC/tests/Utilities/
I thought the tests directory would have not been moved.

chrisburr · 2020-12-07T19:22:30Z

I have seen you moved tests/Utilities → src/DIRAC/tests/Utilities/
I thought the tests directory would have not been moved.

Unfortunately it's a bit messy as some bits of test code are being imported as:

from DIRAC.tests.Utilities import something

There are other ways of fixing this this and I'm happy to do it differently if you have a strong opinion? I think the correct solution is probably to tidy up the tests a bit but that's a task for another PR.

fstagni · 2020-12-07T20:10:02Z

Alright. #4859

andresailer · 2020-12-08T11:44:31Z

There is a lot of documentation out there that points to the location of dirac-install.py where it is at the moment in the integration branch on github, isn't there?

fstagni · 2020-12-08T12:48:35Z

Excellent observation @andresailer

I re-launch: what if we move dirac-install.py in management repo? It has always been a script not depending from any specific release, and, in a way, this PR is also a first step towards retirement of dirac-install script.

chrisburr · 2020-12-08T13:39:53Z

(In the current model of packaging and updating) Is there a use case for dirac-install.py being included in the installation? If not, I think it's a great idea to move it over to DIRACGrid/management so we can have only a single breakage from the URL changing.

atsareg · 2020-12-08T15:00:51Z

dirac-install is used to upgrade the already existing installation. In this case typically it is taken from the installation itself. However, it is the same dirac-install (may be a different version) as the one for an initial installation. So, moving it to management does not harm

chrisburr · 2020-12-08T16:10:14Z

dirac-install is used to upgrade the already existing installation

Won't moving it to management break this workflow as it will no longer exist in DIRAC/Core/scripts? Or is the suggestion to copy it and use the file in management only during manual installations?

fstagni · 2020-12-08T17:11:22Z

Well, we can replace the dirac-install script that's here with a script that just downloads the dirac-install that is in management and run it. This would BTW make sure that we don't run into backward incompatibilities.
There are other solutions of course.

atsareg · 2020-12-08T19:00:51Z

Is the dirac-distribution.py and the related stuff updated in the management project to take this change into account ?

atsareg · 2020-12-08T19:05:53Z

This change is probably not so complex but still can have lots of repercussions here and there. So, I would suggest to make it the first one in v7r3 release. I mean I will make rel-v7r2 branch with the current integration contents which should be just certified for the release. Then I merge this one into integration straight away. What do you think ?

fstagni · 2020-12-09T08:37:09Z

I think that this PR can be merged as it is in v7r2. For the move of dirac-install out of this repo we can wait v7r3.

andresailer · 2020-12-09T08:52:48Z

Can we please keep a copy of dirac-install at its current location, and not move it a week before the end of year break? Maybe add a deprecation print out at the end of the script? Ideally already putting one in diracgrid/management and referring to that location in the previously mentioned deprecation message?

fstagni · 2020-12-09T08:59:03Z

I opened issue #4860 to discuss the "dirac-install move" topic, as that won't be part of this PR. Here, what should be addressed is only this:

There is a lot of documentation out there that points to the location of dirac-install.py where it is at the moment in the integration branch on github, isn't there?

chrisburr · 2020-12-09T10:02:05Z

This change is probably not so complex but still can have lots of repercussions here and there. So, I would suggest to make it the first one in v7r3 release.

I would really like to include this and the entry_points PR. Without them it will be impossible for me to keep maintaining the conda-forge package without considerable work. I also think it will delay DIRAC moving to Python 3 by at least 6 months as we can't start thinking about properly testing extensions without these PRs. I admit it's likely to cause some disruption for the certification hackathons but delaying v7r2 release by a couple of weeks seems preferable to me, especially as no one seems to desperate to use the other features in v7r2?

For the move of dirac-install out of this repo we can wait v7r3.

I agree, I'll add a copy of it now.

atsareg · 2020-12-09T10:09:28Z

I guess I will not be able to compile the v7r2 distribution tar balls if this PR is merged ?

chrisburr · 2020-12-09T10:17:01Z

I guess I will not be able to compile the v7r2 distribution tar balls if this PR is merged ?

@atsareg Can you point me to the docs on exactly how you do this so I can check?

atsareg · 2020-12-09T10:24:16Z

You can see it here https://dirac.readthedocs.io/en/latest/DeveloperGuide/ReleaseProcedure/index.html#deploying-dirac-tarballs

fstagni · 2020-12-09T10:45:59Z

Basically: you would need to modify a line or 2, probably here: https://github.com/DIRACGrid/management/blob/master/dirac-distribution/dirac-create-distribution-tarball.py#L463 (I only had a very quick look) and then the docker container will need to be re-created. To force that, temporary change this line: https://github.com/DIRACGrid/management/blob/master/.github/workflows/images-creator.yml#L14

chrisburr · 2020-12-09T10:50:08Z

I've just opened the PR: DIRACGrid/management#28

I've tested it locally with a hack to make it clone this branch instead of the actual tag and it seems to work.

fstagni · 2020-12-09T10:57:09Z

Yes, merged. The containers are created here: https://github.com/DIRACGrid/management/actions/runs/410422572

chrisburr · 2020-12-09T11:11:42Z

I've checked and the new image works as expected.

@atsareg Can you ping me once you've uploaded the tarballs so I can run some checks before tomorrow's hackathon rather than risking it needing to be cancelled

chrisburr requested review from andresailer, atsareg, chaen and fstagni as code owners December 7, 2020 15:17

chrisburr commented Dec 7, 2020

View reviewed changes

fstagni mentioned this pull request Dec 7, 2020

Move integration tests under /tests #4859

Closed

fstagni mentioned this pull request Dec 9, 2020

Move dirac-install to management repo #4860

Closed

chrisburr added 4 commits December 9, 2020 11:15

Move sources to src/DIRAC/

3c530ef

Fix tests for new layout

1f0aaf4

Fix integration tests pylint

9b3c798

Restore Core/scripts/dirac-install.py

3c474c1

chrisburr force-pushed the src-layout branch from a91888f to 3c474c1 Compare December 9, 2020 10:15

atsareg approved these changes Dec 9, 2020

View reviewed changes

atsareg merged commit ebf6678 into DIRACGrid:integration Dec 9, 2020

chrisburr deleted the src-layout branch December 9, 2020 11:09

andresailer mentioned this pull request Dec 9, 2020

[int] Docs: RTD: fix path to DIRAC source code #4862

Merged

chaen mentioned this pull request Dec 9, 2020

[v7r2] X509Request: correct comparison of public keys #4863

Merged

andresailer mentioned this pull request Dec 21, 2020

Core/scripts/dirac-install.py was removed #4880

Closed

Uh oh!

Conversation

chrisburr commented Dec 7, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chrisburr Dec 7, 2020

Choose a reason for hiding this comment

Uh oh!

chaen Dec 8, 2020

Choose a reason for hiding this comment

Uh oh!

chaen Dec 8, 2020

Choose a reason for hiding this comment

Uh oh!

fstagni commented Dec 7, 2020

Uh oh!

chrisburr commented Dec 7, 2020

Uh oh!

fstagni commented Dec 7, 2020

Uh oh!

andresailer commented Dec 8, 2020

Uh oh!

fstagni commented Dec 8, 2020

Uh oh!

chrisburr commented Dec 8, 2020

Uh oh!

atsareg commented Dec 8, 2020

Uh oh!

chrisburr commented Dec 8, 2020

Uh oh!

fstagni commented Dec 8, 2020

Uh oh!

atsareg commented Dec 8, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

atsareg commented Dec 8, 2020

Uh oh!

fstagni commented Dec 9, 2020

Uh oh!

andresailer commented Dec 9, 2020

Uh oh!

fstagni commented Dec 9, 2020

Uh oh!

chrisburr commented Dec 9, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

atsareg commented Dec 9, 2020

Uh oh!

chrisburr commented Dec 9, 2020

Uh oh!

atsareg commented Dec 9, 2020

Uh oh!

fstagni commented Dec 9, 2020

Uh oh!

chrisburr commented Dec 9, 2020

Uh oh!

fstagni commented Dec 9, 2020

Uh oh!

chrisburr commented Dec 9, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

chrisburr commented Dec 7, 2020 •

edited

Loading

atsareg commented Dec 8, 2020 •

edited

Loading

chrisburr commented Dec 9, 2020 •

edited

Loading